An Information Theoretic Measure of Sequence Recognition Performance Idiap-com 02-03 an Information Theoretic Measure of Sequence Recognition Performance

Sequence recognition performance is often summarised first in terms of the number of hits (H), substitutions (S), deletions (D) and insertions (I), and then as a single statistic by the “word error rate” WER = 100(S+D+I)/(H+S+D). While in common use, WER has two disadvantages as a performance measure. One is that it has no upper bound, so it doesn’t tell you how good a system is, only that one is better than another. The other is that it is not D/I symmetric, although deletions and insertions are equally disadvantageous. At low error rates these limitations can be ignored. However, for the high error rates which can occur during tests for speech recognition in noise the WER measure starts to misbehave, giving far more weight to insertions than to deletions and regularly “exceeding 100%”. Here we derive an alternative summary statistic for sequence recognition accuracy: WIP = H2/(H+S+D)(H+S+I). The WIP (word information preserved) measure results from an approximation to the proportion of the information about the true sequence which is preserved in the recognised sequence. It has comparable simplicity to WER but neither of its disadvantages.